Planning and managing network probes using centralized controller

ABSTRACT

In general, the disclosure describes techniques for measuring edge-based quality of experience (QoE) metrics. For instance, a network device may construct a topological representation of a network, including indications of nodes and links connecting the nodes within the network. For each of the links, the network device may select a node device of the two node devices connected by the respective link to measure one or more QoE metrics for the respective link, with the non-selected node device not measuring the QoE metrics. In response to selecting the selected node device, the network device may receive a set of one or more QoE metrics for the respective link for data flows flowing from the selected node device to the non-selected node device. The network device may store the QoE metrics and determine counter QoE metrics for data flows flowing from the non-selected node device to the selected node device.

TECHNICAL FIELD

The disclosure relates to computer networks.

BACKGROUND

Over the last few decades, the Internet has grown exponentially from asmall network comprising of few nodes to a worldwide pervasive networkthat services more than a billion users. Today, individual subscribersare not limited to running a few network sessions with voice and/or datadownloads over the network. Instead, the extent of services used bysubscribers varies widely from multimedia gaming, audio and videostreaming, web services, voice over IP (VoIP), and the like. With newtechnology penetration, such as increased utilization of Internet ofThings (IoT) and M2M (machine to machine) communications, the networkservices and the software applications that a given subscriber mayrequire also varies from a few sessions to multiple sessions havingconcurrent flows. This number is growing rapidly as subscribersincreasingly run multiple applications, services, transactionssimultaneously. The increased amount and variety of subscriber sessionsand packet flows create challenges for network service providers withrespect to network performance, such as latency, delay, and jitter.

SUMMARY

In general, the disclosure describes techniques for a centralizedcontroller using network topology information to developing a probingplan in accordance with high-level intent configuration to determineapplication quality of experience metrics (QoE) for links in thenetwork. For example, a controller such as a Software-Defined Networking(SDN) controller that implements the techniques described herein mayevaluate a topology of a network such as a Software-Defined Wide AreaNetworks (SD-WAN), determining which nodes are connected and how manyother nodes each node is directly connected to. The SDN controller mayevaluate traffic flows to construct the topology of the network forevaluation. This topology may also indicate which nodes are performingprobing processes on the various links. Using this topology, the SDNcontroller may modify the probing plan implemented in the topology, suchas by pruning from the topology to cease some of the probing processesthat the SDN controller determines to be redundant, reducing the overalltraffic in the network.

In some SD-WANs, the SDN controller may specify a path for data flowsbetween client devices and application servers. These paths aretypically selected using service-level agreement (SLA) parameters andvarious QoE metrics of the WAN links. While the SLA parameters may bemore static in nature, or at least predetermined prior to an SD-WANappliance receiving the flow, the metrics of the various WAN links maybe more dynamic, as the metrics describing the capabilities of theparticular WAN link may vary based on various current aspects of thenetwork. These metrics are obtained by sending probe packets on thevarious links and analyzing the results of the transmission, where probepackets having the same size as the data packets in the data flowreasonably measure how the particular WAN link could handle the dataflow.

The probe packets that are sent over the links within a network may takeup valuable bandwidth and other resources, as a node (e.g., an SD-WANappliance) is inserting data into the flows that are in addition to thenormal application traffic that must be transmitted over the links. Wheneach node device is transmitting multiple probe packets over every linkconnected to the respective node, the amount of extraneous datatransmitted over the links can be great. This issue is only compoundedwhen node devices send probe packets over complex paths consisting ofmultiple links in an effort to determine QoE metrics for the complexpath.

Rather than instructing each node device to measure the QoE metrics foreach link connected to the node device and/or for each other node devicein the SD-WAN, including over complex paths, the techniques describedherein may select a single node of the two nodes connected by aparticular link to measure the QoE metrics for the particular link. TheSDN controller may then cycle through the links and select one of thetwo nodes connected by the respective link to measure the QoE metricsfor that particular link. From this measured set of QoE metrics, the SDNcontroller may extrapolate the data to estimate QoE metrics for thereverse direction on the same link and for complex paths consisting ofmultiple links, including the link for which the QoE metrics weremeasured. For instance, the SDN controller may use the measured QoEmetrics as the QoE metrics for the reverse direction of the link (e.g.,the direction flowing from the non-selected node device to the selectednode device).

There may be one or more advantages to using the techniques describedherein. While the actual metrics may not be identical in all situations,a particular link will generally have similar QoE metrics in eachdirection. Similarly, when sending probe packets over a complex pathconsisting of multiple links, while the QoE metrics may not exactly be acombination of the links that make up the complex path, taking acombination of the QoE metrics across each link may still provideaccurate estimates of the QoE metrics for the path. As such, estimatingthe QoE metrics for various links between node devices in the mannerdescribed by the techniques of this disclosure may greatly reduce thebandwidth and other resources consumed by the node devices in an effortto reliably determine QoE metrics for the entire network. As nodedevices have fewer probe packets to generate and process, this may alsoincrease the efficiency of each individual node device, includingreducing central processing unit (CPU) usage, memory using, and powerconsumption for each individual node device, thereby improving thefunctioning of the node devices.

Further, the SDN controller may capture the user intent (e.g., tomeasure user diagram protocol (UDP) performance across the network viasynthetic probes). The SDN controller may also analyze the networktopology and build a map of the current topology, all while utilizingexisting probes. The SDN controller may reduce redundancy by using asingle measurement to extrapolate QoE metrics for reverse paths andadditional logical paths (e.g., VPNs, etc.) originating from the samedevice. The SDN controller may prune the probes to avoid duplicates andredundancy (e.g., device on the reverse path need not probe again). TheSDN controller may alternatively or additionally redistribute the probesto ensure the uniform distribution across the topology (or the sub-setwithin the topology). In other words, given the potential non-uniformityof network topologies, the SDN controller may execute an algorithm thatredistributes the probing responsibilities across the various nodedevices such that no single node device is tasked with probing a vastlyunequal amount of links.

The SDN controller may leverage the existing interface on the device toconfigure and monitor the probes. This architecture may further maintainthis intent map to correlate the probe related metrics (e.g., jitter,packet loss, etc.). Through utilizing this SDN controller, a networkadministrator may better visualize the state of network to help aid inbetter planning.

In one example of the techniques described herein, a method isdescribed, the method including, constructing, by a network device for anetwork that includes a plurality of node devices, a topologicalrepresentation of the network, wherein the topological representationcomprises an indication of each of the plurality of node devices and anindication of each link of a plurality of links, each link connectingtwo node devices of the plurality of node devices. The method alsoincludes, for each of the plurality of links, selecting, by the networkdevice and based on the topological representation of the network, anode device of the two node devices connected by the respective link tomeasure one or more quality of experience (QoE) metrics for therespective link, wherein the non-selected node device does not measurethe QoE metrics for the respective link, in response to selecting theselected node device to measure the one or more QoE metrics for therespective link, receiving, by the network device and from the selectednode device, a set of one or more QoE metrics for the respective link,wherein the set of one or more QoE metrics indicate QoE metrics for dataflows flowing from the selected node device to the non-selected nodedevice, storing, by the network device, the set of QoE metrics for therespective link in a database, and determining, by the network deviceand based on the set of one or more QoE metrics for the respective link,a set of one or more counter QoE metrics indicating QoE metrics for dataflows flowing from the non-selected node device to the selected nodedevice.

In another example of the techniques described herein, a network deviceis described. The network device includes a memory. The network devicealso includes one or more processors in communication with the memory.The one or more processors are configured to, construct, for a networkthat includes a plurality of node devices, a topological representationof the network, wherein the topological representation comprises anindication of each of the plurality of node devices and an indication ofeach link of a plurality of links, each link connecting two node devicesof the plurality of node devices. The one or more processors are alsoconfigured to, for each of the plurality of links, select, based on thetopological representation of the network, a node device of the two nodedevices connected by the respective link to measure one or more qualityof experience (QoE) metrics for the respective link, wherein thenon-selected node device does not measure the QoE metrics for therespective link, in response to selecting the selected node device tomeasure the one or more QoE metrics for the respective link, receive,from the selected node device, a set of one or more QoE metrics for therespective link, wherein the set of one or more QoE metrics indicate QoEmetrics for data flows flowing from the selected node device to thenon-selected node device, store the set of QoE metrics for therespective link in a database, and determine, based on the set of one ormore QoE metrics for the respective link, a set of one or more counterQoE metrics indicating QoE metrics for data flows flowing from thenon-selected node device to the selected node device.

In another example of the techniques described herein, a non-transitorycomputer-readable storage medium is described, the non-transitorycomputer-readable storage medium storing instructions thereon that whenexecuted cause one or more processors, via execution of asoftware-defined networking (SDN) device, to, construct, for a networkthat includes a plurality of node devices, a topological representationof the network, wherein the topological representation comprises anindication of each of the plurality of node devices and an indication ofeach link of a plurality of links, each link connecting two node devicesof the plurality of node devices. The instructions, when executed, alsocause the one or more processors to, for each of the plurality of links,select, based on the topological representation of the network, a nodedevice of the two node devices connected by the respective link tomeasure one or more quality of experience (QoE) metrics for therespective link, wherein the non-selected node device does not measurethe QoE metrics for the respective link, in response to selecting theselected node device to measure the one or more QoE metrics for therespective link, receive, from the selected node device, a set of one ormore QoE metrics for the respective link, wherein the set of one or moreQoE metrics indicate QoE metrics for data flows flowing from theselected node device to the non-selected node device, store the set ofQoE metrics for the respective link in a database, and determine, basedon the set of one or more QoE metrics for the respective link, a set ofone or more counter QoE metrics indicating QoE metrics for data flowsflowing from the non-selected node device to the selected node device.

In another example of the techniques described herein, a method isdescribed, the method including, receiving, by a network device,configuration data indicative of a user intent for measuring applicationquality of experience (QoE) in a network that includes a plurality ofnode devices and a plurality of links, each link connecting two nodedevices of the plurality of node devices. The method further includesconstructing, by the network device, a topological representation of thenetwork, wherein the topological representation comprises an indicationof each of the plurality of node devices and an indication of each linkof the plurality of links. The method also included modifying, by thenetwork device and based on the user intent and the topologicalrepresentation of the network, one or more entries from an initialprobing list to create a modified probing list, wherein the initialprobing list comprises a plurality of entries, wherein each entryindicates a particular node device from the plurality of node that isperforming a probing process for a particular link of the plurality oflinks. The method further includes instructing, by the network deviceand in accordance with the modified probing list, one or more nodedevices of the plurality of node devices to perform the probing processon the respective link in the corresponding entry in the modifiedprobing list, wherein the probing process generates the application QoEmetric data for the respective link. The method also includes, inresponse to receiving the application QoE metric data from the one ormore node devices instructed to perform the probing process,aggregating, by the network device, the application QoE metric data.

In another example of the techniques described herein, a network deviceis described. The network device includes a memory. The network devicealso includes one or more processors in communication with the memory.The one or more processors are configured to receive configuration dataindicative of a user intent for measuring application quality ofexperience (QoE) in a network that includes a plurality of node devicesand a plurality of links, each link connecting two node devices of theplurality of node devices. The one or more processors are furtherconfigured to construct a topological representation of the network,wherein the topological representation comprises an indication of eachof the plurality of node devices and an indication of each link of theplurality of links. The one or more processors are also configured tomodify, based on the user intent and the topological representation ofthe network, one or more entries from an initial probing list to createa modified probing list, wherein the initial probing list comprises aplurality of entries, wherein each entry indicates a particular nodedevice from the plurality of node that is performing a probing processfor a particular link of the plurality of links. The one or moreprocessors are further configured to instruct, in accordance with themodified probing list, one or more node devices of the plurality of nodedevices to perform the probing process on the respective link in thecorresponding entry in the modified probing list, wherein the probingprocess generates the application QoE metric data for the respectivelink. The one or more processors are also configured to, in response toreceiving the application QoE metric data from the one or more nodedevices instructed to perform the probing process, aggregate theapplication QoE metric data.

In another example of the techniques described herein, a non-transitorycomputer-readable storage medium is described, the non-transitorycomputer-readable storage medium storing instructions thereon that whenexecuted cause one or more processors, via execution of asoftware-defined networking (SDN) device, to receive configuration dataindicative of a user intent for measuring application quality ofexperience (QoE) in a network that includes a plurality of node devicesand a plurality of links, each link connecting two node devices of theplurality of node devices. The instructions further cause the one ormore processors to construct a topological representation of thenetwork, wherein the topological representation comprises an indicationof each of the plurality of node devices and an indication of each linkof the plurality of links. The instructions also cause the one or moreprocessors to modify, based on the user intent and the topologicalrepresentation of the network, one or more entries from an initialprobing list to create a modified probing list, wherein the initialprobing list comprises a plurality of entries, wherein each entryindicates a particular node device from the plurality of node that isperforming a probing process for a particular link of the plurality oflinks. The instructions further cause the one or more processors toinstruct, in accordance with the modified probing list, one or more nodedevices of the plurality of node devices to perform the probing processon the respective link in the corresponding entry in the modifiedprobing list, wherein the probing process generates the application QoEmetric data for the respective link. The instructions also cause the oneor more processors to, in response to receiving the application QoEmetric data from the one or more node devices instructed to perform theprobing process, aggregate the application QoE metric data.

The details of one or more examples are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example software-defined widearea network system that performs edge-based routing techniques, inaccordance with the techniques of this disclosure.

FIG. 2 is a block diagram illustrating an example software-defined widearea network, in accordance with one or more techniques of thisdisclosure.

FIG. 3 is a block diagram illustrating an example software-defined widearea network controller configured to execute one or more functions toperform edge-based routing techniques, in accordance with the techniquesof this disclosure.

FIG. 4 is a block diagram illustrating an example software-defined widearea network appliance configured to execute one or more functions toperform edge-based routing techniques, in accordance with the techniquesof this disclosure.

FIG. 5 is a conceptual flow diagram illustrating an example topology fora plurality of nodes in a software-defined wide area network system, inaccordance with the techniques of this disclosure.

FIG. 6 is conceptual diagram illustrating an example architecture of acloud controller configured to perform edge-based routing techniques, inaccordance with the techniques of this disclosure.

FIG. 7 is a flow diagram illustrating an example technique for asoftware-defined wide area network system that performs edge-basedrouting functions for nodes in the network, in accordance with thetechniques of this disclosure.

FIG. 8 is a flow diagram illustrating an example technique for asoftware-defined wide area network system that performs edge-basedrouting functions for nodes in the network, in accordance with thetechniques of this disclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example software-defined widearea network system 2 that performs edge-based routing techniques, inaccordance with the examples described herein.

The example network system of FIG. 1 includes a SD-WAN system 2 thatoperates as a private network to provide packet-based network servicesto subscriber devices 16. That is, SD-WAN system 2 providesauthentication and establishment of network access for subscriberdevices 16 such that a subscriber device may begin exchanging datapackets with public network 12, which may be an internal or externalpacket-based network such as the Internet.

In the example of FIG. 1, SD-WAN system 2 comprises access network 6that provides connectivity to public network 12 via service providersoftware-defined wide area network 7 (hereinafter, “SD-WAN 7”) androuter 8. SD-WAN 7 and public network 12 provide packet-based servicesthat are available for request and use by subscriber devices 16.

As examples, SD-WAN 7 and/or public network 12 may provide bulk datadelivery, voice over Internet protocol (VoIP), Internet Protocoltelevision (IPTV), Short Messaging Service (SMS), Wireless ApplicationProtocol (WAP) service, or customer-specific application services.Public network 12 may comprise, for instance, a local area network(LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), anenterprise LAN, a layer 3 virtual private network (VPN), an InternetProtocol (IP) intranet operated by the service provider that operatesaccess network 6, an enterprise IP network, or some combination thereof.In various examples, public network 12 is connected to a public WAN, theInternet, or to other networks. Public network 12 executes one or morepacket data protocols (PDPs), such as IP (IPv4 and/or IPv6), X.25 orPoint-to-Point Protocol (PPP), to enable packet-based transport ofpublic network 12 services.

In general, subscriber devices 16 connect to gateway router 8 via accessnetwork 6 to receive connectivity to subscriber services forapplications hosted by public network 12 or router 9. A subscriber mayrepresent, for instance, an enterprise, a residential subscriber, or amobile subscriber. Subscriber devices 16 may be, for example, personalcomputers, laptop computers or other types of computing devicespositioned behind customer equipment (CE) 11, which may provide localrouting and switching functions. Each of subscriber devices 16 may run avariety of software applications, such as word processing and otheroffice support software, web browsing software, software to supportvoice calls, video games, video conferencing, and email, among others.For example, subscriber device 16 may be a variety of network-enableddevices, referred generally to as “Internet-of-Things” (IoT) devices,such as cameras, sensors (S), televisions, appliances, etc. In addition,subscriber devices 16 may comprise mobile devices that access the dataservices of SD-WAN system 2 via a radio access network (RAN) 6. Examplemobile subscriber devices include mobile telephones, laptop or desktopcomputers having, e.g., a 3G wireless card, wireless-capable netbooks,video game devices, pagers, smart phones, personal data assistants(PDAs) or the like.

A network service provider operates, or in some cases leases, elementsof access network 6 to provide packet transport between subscriberdevices 16 and router 8. Access network 6 represents a network thataggregates data traffic from one or more of subscriber devices 16 fortransport to/from SD-WAN 7 of the service provider. Access network 6includes network nodes that execute communication protocols to transportcontrol and user data to facilitate communication between subscriberdevices 16 and router 8. Access network 6 may include a broadband accessnetwork, a wireless LAN, a public switched telephone network (PSTN), acustomer premises equipment (CPE) network, or other type of accessnetwork, and may include or otherwise provide connectivity for cellularaccess networks, such as a radio access network (RAN) (not shown).Examples include networks conforming to a Universal MobileTelecommunications System (UMTS) architecture, an evolution of UMTSreferred to as Long Term Evolution (LTE), mobile IP standardized by theInternet Engineering Task Force (IETF), as well as other standardsproposed by the 3^(rd) Generation Partnership Project (3GPP), 3^(rd)Generation Partnership Project 2 (3GGP/2) and the WiMAX forum.

SD-WAN appliance 18 may be a customer edge (CE) router, a provider edge(PE) router, or other network device between access network 6 and SD-WAN7. SD-WAN 7 offers packet-based connectivity to subscriber devices 16attached to access network 6 for accessing public network 12 (e.g., theInternet). SD-WAN 7 may represent a public network that is owned andoperated by a service provider to interconnect a plurality of networks,which may include access network 6. In some examples, SD-WAN 7 mayimplement Multi-Protocol Label Switching (MPLS) forwarding and in suchinstances may be referred to as an MPLS network or MPLS backbone. Insome instances, SD-WAN 7 represents a plurality of interconnectedautonomous systems, such as the Internet, that offers services from oneor more service providers. Public network 12 may represent the Internet.Public network 12 may represent an edge network coupled to SD-WAN 7 viaa transit network 22 and one or more network devices, e.g., a customeredge device such as customer edge switch or router. Public network 12may include a data center. Router 8 may exchange packets with servicenodes 10 via virtual network 20, and router 8 may forward packets topublic network 12 via transit network 22.

In examples of network 2 that include a wireline/broadband accessnetwork, router 8 may represent a Broadband Network Gateway (BNG),Broadband Remote Access Server (BRAS), MPLS PE router, core router orgateway, or Cable Modem Termination System (CMTS). In examples ofnetwork 2 that include a cellular access network as access network 6,router 8 may represent a mobile gateway, for example, a Gateway GeneralPacket Radio Service (GPRS) Serving Node (GGSN), an Access Gateway(aGW), or a Packet Data Network (PDN) Gateway (PGW). In other examples,the functionality described with respect to router 8 may be implementedin a switch, service card or another network element or component. Insome examples, router 8 may itself be a service node.

A network service provider that administers at least parts of network 2typically offers services to subscribers associated with devices, e.g.,subscriber devices 16, that access SD-WAN system 2. Services offered mayinclude, for example, traditional Internet access, VoIP, video andmultimedia services, and security services. As described above withrespect to SD-WAN 7, SD-WAN 7 may support multiple types of accessnetwork infrastructures that connect to service provider network accessgateways to provide access to the offered services. In some instances,the network system may include subscriber devices 16 that attach tomultiple different access networks 6 having varying architectures.

In general, any one or more of subscriber devices 16 may requestauthorization and data services by sending a session request to agateway device such as SD-WAN appliance 18 or router 8. In turn,software-defined wide area network (“SD-WAN”) appliance 18 may access acentral server (not shown) such as an Authentication, Authorization andAccounting (AAA) server to authenticate the one of subscriber devices 16requesting network access. Once authenticated, any of subscriber devices16 may send subscriber data traffic toward SD-WAN 7 to access andreceive services provided by public network 12, and such packets maytraverse router 8 as part of at least one packet flow. In some examples,SD-WAN appliance 18 may forward all authenticated subscriber traffic topublic network 12, and router 8 may apply services 15 and/or steerparticular subscriber traffic to a data center 9 if the subscribertraffic requires services on service nodes 10. Applications (e.g.,service applications) to be applied to the subscriber traffic may behosted on service nodes 10.

For example, when forwarding subscriber traffic, router 8 may directindividual subscriber packet flows through services 15 executing on oneor more service cards installed within router 9. In addition, oralternatively, SD-WAN system 2 includes a data center 9 having a clusterof service nodes 10 that provide an execution environment for the mostlyvirtualized network services. In some examples, each of service nodes 10represents a service instance. Each of service nodes 10 may apply one ormore services to traffic flows. As such, router 8 may steer subscriberpacket flows through defined sets of services provided by service nodes10. That is, in some examples, each subscriber packet flow may beforwarded through a particular ordered combination of services providedby service nodes 10, each ordered set being referred to herein as a“service chain.” As examples, services 15 and/or service nodes 10 mayapply stateful firewall (SFW) and security services, deep packetinspection (DPI), carrier grade network address translation (CGNAT),traffic destination function (TDF) services, media (voice/video)optimization, Internet Protocol security (IPSec)/virtual private network(VPN) services, hypertext transfer protocol (HTTP) filtering, counting,accounting, charging, and/or load balancing of packet flows, or othertypes of services applied to network traffic.

In the example of FIG. 1, subscriber packet flows may be directed alonga service chain that includes any of services 15 and/or services appliedby service nodes 10. Once processed at a terminal node of the servicechain, i.e., the last service to be applied to packets flowing along aparticular service path, the traffic may be directed to public network12.

Whereas a “service chain” defines one or more services to be applied ina particular order to provide a composite service for application topacket flows bound to the service chain, a “service tunnel” or “servicepath” refers to a logical and/or physical path taken by packet flowsprocessed by a service chain along with the forwarding state forforwarding packet flows according to the service chain ordering. Eachservice chain may be associated with a respective service tunnel, andpacket flows associated with each subscriber device 16 flow alongservice tunnels in accordance with a service profile associated with therespective subscriber. For example, a given subscriber may be associatedwith a particular service profile, which in turn is mapped to a servicetunnel associated with a particular service chain. Similarly, anothersubscriber may be associated with a different service profile, which inturn is mapped to a service tunnel associated with a different servicechain. In some examples, after SD-WAN appliance 18 has authenticated andestablished access sessions for the subscribers, SD-WAN appliance 18 orrouter 8 may direct packet flows for the subscribers along theappropriate service tunnels, thereby causing data center 9 to apply therequisite ordered services for the given subscriber. In some examples,SDN controller 14 may also provide a forwarding rule set to SD-WANappliance 18 or router 8 for managing the forwarding path. In someexamples, SDN controller 14 manages the forwarding path through allelements in data center 9 starting at router 8.

In some examples, service nodes 10 may implement service chains usinginternally configured forwarding state that directs packets of thepacket flow along the service chains for processing according to theidentified set of service nodes 10. Such forwarding state may specifytunnel interfaces for tunneling between service nodes 10 using networktunnels such as IP or Generic Route Encapsulation (GRE) tunnels, NetworkVirtualization using GRE (NVGRE), or by using VLANs, Virtual ExtensibleLANs (VXLANs), MPLS techniques, and so forth. In some instances, real orvirtual switches, routers or other network elements that interconnectservice nodes 10 may be configured to direct the packet flow to theservice nodes 10 according to service chains.

In the example of FIG. 1, SD-WAN system 2 comprises a software definednetwork (SDN) and network functions virtualization (NFV) architecture.SDN controller device 14 may provide a high-level controller forconfiguring and managing the routing and switching infrastructure ofSD-WAN system 2. While some instances described herein relate to SD-WANappliance 18 performing the edge-based routing techniques describedherein, SDN controller 14 may also perform these techniques for SD-WANsystem 2. NFV orchestrator device 13 may provide a high-levelorchestrator for configuring and managing virtualization of networkservices into service nodes 10 of data center 9. In some instances, SDNcontroller 14 manages deployment of virtual machines (VMs) within theoperating environment of data center 9. For example, SDN controller 14may interact with provider edge (PE) router 8 to specify service chaininformation, described in more detail below. For example, the servicechain information provided by SDN controller 14 may specify anycombination and ordering of services provided by service nodes 10,traffic engineering information for tunneling or otherwise transportingpacket flows along service paths, rate limits, Type of Service (TOS)markings or packet classifiers that specify criteria for matching packetflows to a particular service chain. Further example details of an SDNcontroller are described in PCT International Patent Application PCT/US13/44378, filed Jun. 5, 2013, the entire content of which isincorporated herein by reference.

Although illustrated as part of data center 9, service nodes 10 may benetwork devices coupled by one or more switches or virtual switches ofSD-WAN 7. In one example, each of service nodes 10 may run as VMs in avirtual compute environment. Moreover, the compute environment maycomprise a scalable cluster of general computing devices, such as x86processor-based servers. As another example, service nodes 10 maycomprise a combination of general purpose computing devices and specialpurpose appliances. As virtualized network services, individual networkservices provided by service nodes 10 can scale just as in a modern datacenter through the allocation of virtualized memory, processorutilization, storage and network policies, as well as horizontally byadding additional load-balanced VMs. In other examples, service nodes 10may be gateway devices or other routers. In further examples, thefunctionality described with respect to each of service nodes 10 may beimplemented in a switch, service card, or another network element orcomponent.

As described herein, elements within SD-WAN system 2, such as SD-WANappliance 18, may perform application data monitoring using variousapplication quality of experience (QoE) metric functions, such asreal-time performance monitoring (RPM) or two-way active measurementprotocol (TWAMP), for example. That is, RPM and TWAMP may be used withinSD-WAN system 2 to measure both one-way and two-way or round-tripmetrics of network performance, such as path connectivity, path delay,packet jitter, packet loss, packet re-ordering, and the like, e.g., on aper-subscriber basis between network devices, also referred to as hostsor endpoints. In general, a QoE measurement architecture includesnetwork devices that each support the used protocol and perform specificroles to start data sessions and exchange test packets for the datasessions. In the example network architecture illustrated in FIG. 1,SD-WAN appliance 18 is configured to perform the QoE metric predictions.SD-WAN appliance 18 allows for load sharing across connections andadjusts traffic flows based on network conditions to improveperformance. Although only a single SD-WAN appliance 18 is shown in FIG.1, in some examples system 2 may include multiple SD-WAN appliances.

Example QoE metric functions include active or synthetic probingperformed by SD-WAN appliance 18. Active or synthetic probing results inadditional traffic, which also varies based on the network topology andtype of packet (e.g., Internet Control Message Protocol (ICMP), UserDatagram Protocol (UDP), Transmission Control Protocol (TCP), HypertextTransfer Protocol (HTTP), HTTP Secure (HTTPS), or an applicationspecific probe). This would mean that on a customer-premises equipment(CPE) kind of deployment, the CPE (e.g., CE 11) may endure aconsiderable amount of these probe packets flooding the network. Thissituation can be worse in the case of a full mesh network topology wherethere may be multiple paths to traverse from one point to another. Asthe probes are scattered across the network, there may not be anycentralized management.

Further, network administrators may need to configure these probepackets. Complex network topologies may produce many redundant probes inthese networks. Worsening this issue, duplicate probes could originatefrom different VPNs of the same CPE device. Such an intent basedconfiguration as the techniques described herein may provide reliablemeasurements and may avoid duplication of data. Implementing thetechniques described herein by a cloud-based controller may optimallyuse a combination of active and passive probes, optimally manage theprobes, and avoid redundant probes.

The techniques described herein may distribute the probes in the case ofa complex network topology. The techniques described herein may alsoprovide a cloud-based correlation of probes. The techniques may alsoprovide a centralized aggregation of probes that are easier to plan andmanage. Further, the techniques described herein may be highly availableand use a fault tolerant management system, leveraging existingfunctionalities.

SD-WAN appliance 18, which performs the edge-based routing algorithms,also determine QoE metrics, such as service level agreement (SLA)metrics that include round-trip time (RTT), jitter, and packet loss,which were influenced by applications' real-time parameters like packetsize, queues and burst of packets to determine the best path. While someinstances described herein relate to SDN controller 14 performing theedge-based routing techniques described herein, SD-WAN appliance 18 mayalso perform these techniques for SD-WAN system 2.

Rather than measuring QoE metrics bi-directionally for every linkbetween nodes and for every shortest path between two nodes of servicenodes 10 in the network, SDN controller 14 may control each of servicenodes 10 to only measure various QoE metrics for each edge between nodesuni-directionally. Further, SDN controller 14 may avoid redundant probesby aggregating the uni-directionally computed QoE metrics to determineQoE metrics for complex paths consisting of multiple edges or for thereverse flow of data across the edges. Active or synthetic probingresults in additional traffic, which also varies based on the networktopology and type of packet. This situation can be worse in the case ofa full mesh network topology where there may be multiple paths totraverse from one point to another. As described below with respect toFIGS. 3-8, the techniques described herein allow SDN controller 14 (orSD-WAN appliance 18) to implement edge-based routing to more accuratelymeasure the various QoE metrics in the network while limiting the numberof probes injected into the system.

In accordance with the techniques described herein, SDN controller 14may be a cloud controller for SD-WAN system 2, coordinating and mappingprobing processes across SD-WAN system 2. For instance, SDN controller14 may construct, for SD-WAN system 2 that includes a plurality of nodedevices (e.g., subscriber devices 16 and service nodes 10), atopological representation of SD-WAN system 2. The topologicalrepresentation may take any form (e.g., a matrix, a database, a graphic,text, or any other data structure) that provides an indication of eachof the node devices and an indication of each link of a plurality oflinks, where each link connects two of the node devices.

In the example of FIG. 1, SN controller 14 may be creating thetopological representation of any number of the networks described inSD-WAN system 2. For instance, the topological representation could beof CEs 11 in access network 6, of subscriber devices 16, of one or moreinstances of SD-WAN appliance 18 in SD-WAN 7, or any combination thereofin an SDN system (e.g., SD-WAN system 2).

For each of the plurality of links, SDN controller 14 may select, basedon the topological representation of the network, a node device of thetwo node devices connected by the respective link to measure one or morequality of experience (QoE) metrics for the respective link. As aresult, the non-selected node device does not measure the QoE metricsfor the respective link, meaning that only the selected node device willperform the active or synthetic probing functions for the respectivelink. In response to selecting the selected node device to measure theone or more QoE metrics for the respective link, SDN controller 14 mayreceive, from the selected node device, a set of one or more QoE metricsfor the respective link. This set of one or more QoE metrics wouldindicate QoE metrics for data flows flowing from the selected nodedevice to the non-selected node device. SDN controller 14 may store theset of QoE metrics for the respective link in a database, such that theQoE metrics for this link may be referenced in further extrapolationsand estimations for other links and/or the reverse direction for thelink. For instance, SDN controller 14 may determine, based on the set ofone or more QoE metrics for the respective link, a set of one or morecounter QoE metrics indicating QoE metrics for data flows flowing fromthe non-selected node device to the selected node device.

Ultimately, these techniques may enable SDN controller 14 to create adynamic representation of the QoE metrics for each link and eachpossible connection, in all directions, in a database. For instance, SDNcontroller 14 may store the received measurements for each link in thedatabase. Then, for each extrapolation/estimation (e.g., for eachreverse direction of the received QoE metrics, for each logical pathconnecting two node devices also connected by a link, and for eachcomplex path that is a shortest path between two node devices notconnected directly by a link), SDN controller 14 may store therespective counter QoE metrics in the database. This information may beupdated as new metrics are received by SDN controller 14. With thisinformation, whenever SDN controller 14 must make a routing decision forapplication traffic or for traffic between nodes, SDN controller 14 mayreference the database and the QoE metrics stored within the database tomake routing decisions for the traffic within the network.

SDN controller 14 may extrapolate these QoE metrics further than simplythe reverse direction for the link. In some examples, SDN controller 14may also estimate QoE metrics for different logical paths, such asvirtual private networks (VPNs), between two node devices where aphysical link is already being measured. Further, for a complex pathconsisting of multiple links, SDN controller 14 may estimate QoE metricsfor the complex path by taking a combination of the QoE metricspreviously measured on the links making up the complex path.

In some examples, the metrics carried by QoE probe packets may includeone or more of timestamps for sending or receiving a test packet, errorestimates for sending or receiving the test packet, a sequence numberfor sending the test packet, a time-to-live (TTL) value for the testpacket, a keepalive packet data unit (PDU), and/or a count of servicedpackets, bytes, or subscribers. The one-way and two-way networkperformance measurements may include keepalive or path connectivity,round trip time (RTT), path delay, packet jitter, packet re-ordering,packet loss, service latency measurements, or service load measurementsbased on the received metrics.

FIG. 2 is a block diagram illustrating an example SD-WAN 37, inaccordance with one or more techniques of this disclosure. In theexample described herein, SD-WAN 7 includes three different WAN links: afirst WAN link 40 coupling SD-WAN appliance 38 to a Multi-Protocol LayerSwitching (MPLS) network 50, a second WAN link 42 coupling SD-WANappliance 38 to Internet 52, and a third WAN link 44 coupling SD-WANappliance 38 to long-term evolution (LTE) network 54. In other examples,SD-WAN 7 may include any number of links of any suitable type fortransmitting data flows between the client side (e.g., client device 36and SD-WAN appliance 38) and the application side (e.g., SD-WANappliance 56 and application server 58).

FIG. 3 is a block diagram illustrating the example SDN controller 14 ofFIG. 1 configured to implement the techniques described herein. In theexample of FIG. 3, SDN controller 60 creates a topologicalrepresentation of the network that contains or is managed by SDNcontroller 60 in order to efficiently create an overall process forprobing the various links within the network. SDN controller 60 mayoperate as a network services controller for a service provider network.In the illustrated example of FIG. 3, SDN controller 60 includes amanagement unit 65, a control unit 64 for controlling operation of SDNcontroller 60, and a network interface 66 for exchanging packets withnetwork devices by inbound link 67 and outbound link 68.

In some examples, control unit 64 and/or management unit 65 may beimplemented as one or more processes executing on one or more virtualmachines of one or more physical computing devices. That is, whilegenerally illustrated and described as executing on a single SDNcontroller 60, aspects of each of these units may be delegated to ordistributed across other computing devices.

Each of control unit 64 and/or management unit 65 may include one ormore processors (not shown) that execute software instructions, such asthose used to define a software or computer program, stored to acomputer-readable storage medium (not shown), such as non-transitorycomputer-readable mediums including a storage device (e.g., a diskdrive, or an optical drive) or a memory (such as Flash memory or RAM) orany other type of volatile or non-volatile memory, that storesinstructions to cause the one or more processors to perform thetechniques described herein. Alternatively, or additionally, each ofcontrol unit 64 and/or management unit 65 may comprise dedicatedhardware, such as one or more integrated circuits, one or moreapplication-specific integrated circuits (ASICs), one or moreApplication Specific Special Processors (ASSPs), one or more FPGAs, orany combination of one or more of the foregoing examples of dedicatedhardware, for performing the techniques described herein. Thearchitecture of SDN controller 60 illustrated in FIG. 3 is shown forexample purposes only and should not be limited to this architecture. Inother examples, SDN controller 60 may be implemented in a variety ofways, such software only, hardware only, or a combination of bothsoftware and hardware.

Management unit 65 may comprise a management layer of SDN controller 60,whereas control unit 64 may comprise a control layer of SDN controller60. Management unit 65 includes an analytics unit 61 and a configurationunit 62. Analytics unit 61 may capture information from physical and/orvirtual network elements within SD-WAN system 2, e.g., a gateway,service nodes 10, or of each data center 9 of FIG. 1, and analyze theinformation for use in managing the network services offered by theservice provider. The information may include statistics, logs, events,and errors.

Configuration unit 62 stores configuration information for the networkelements within SD-WAN system 2. In some examples, the configurationinformation comprises a virtual network configuration. Configurationunit 62 may translate a high-level data model of the intended virtualnetwork configuration to a lower-level data model for use in interactingwith the network elements.

Control unit 64 of SDN controller 60 implements a centralized controlplane for SD-WAN system 2 that is responsible for maintaining aconstantly changing network state. Control unit 64 interacts with thenetwork elements within SD-WAN system 2 to maintain a consistent networkstate across all of the network elements. Control unit 64 provides anoperating environment for a command line interface daemon 75 (“CLI 75”)that provides an interface by which an administrator or other managemententity may modify the configuration of SDN controller 60 usingtext-based commands. Control unit 64 also provides an operatingenvironment for several protocols 70, including Border Gateway Protocol(BGP) 72 and Extensible Messaging and Presence Protocol (XMPP) 74 asillustrated in the example of FIG. 3. In accordance with the techniquesdescribed herein, these commands may alter which QoE metrics are to bemeasured, how SDN controller 60 constructs the topology of the network,and whether SDN controller 60 optimizes the distribution of the probingprocesses being performed across the node devices. The userconfiguration (e.g., intent-based configuration) may also configure theprobes themselves, providing instructions as to whether the probes areactive probes, passive probes, or a combination thereof.

In some examples, control unit 64 uses XMPP 74 to communicate withnetwork elements within SD-WAN system 2, such as gateways 8, clientdevices 16, or service nodes 10 of data center 9 within SD-WAN system 2of FIG. 1, by an XMPP interface (not shown). Virtual network route data,statistics collection, logs, and configuration information may be sentas extensible markup language (XML) documents in accordance with XMPP 74for communication between SDN controller 60 and the network elements.Control unit 64 may also use XMPP 74 to communicate with one or both ofanalytics unit 61 and configuration unit 62 of SDN controller 60.

Control unit 64 further includes device manager 71 topology unit 76,probe management unit 77, topology database 78, and metric database 79,which enables control unit 64 to construct a topological representationof the nodes, links, and probing processes to develop a full probingplan across the topology of the network. In accordance with thetechniques described herein, SDN controller 60 may be a cloud controllerfor a network, coordinating and mapping probing processes across thenetwork. For instance, topology unit 76 may construct, for the networkthat includes a plurality of node devices (e.g., subscriber devices andservice nodes), a topological representation of the network. Thetopological representation may take any form (e.g., a matrix, adatabase, a graphic, text, or any other data structure) that provides anindication of each of the node devices and an indication of each link ofa plurality of links, where each link connects two of the node devices.The topological representation may be stored in topology database 78.

In some examples, topology unit 76 may translate high-level data modelsassociated with a topology of the network into lower-level modelssuitable for interacting with network elements or devices, such as thenetwork devices shown in FIGS. 1 and 2. In some cases, topology unit 76may receive, via network interface 66, high-level data models (e.g.,user intent-based network or data models) from an orchestration engineand/or an administrator. These models may be associated with a topologyof a network. Topology unit 76 may use these models and intent toconstruct the topological representation to store within topologydatabase 78.

In constructing the topological representation of the network, topologyunit 76 may monitor one or more probe packets sent over each of theplurality of links. For instance, when the techniques described hereinare first applied to the network, each node device may be sending probepackets over each of the node device's respective links. However,topology unit 76 may not inherently possess a data structure thatdepicts which node devices are in the network, and which other nodedevices each node device is linked to. By monitoring the various probepackets sent across the network, topology unit 76 may populate a datastructure, such as an adjacency matrix, with an indication of which nodedevices are in the network, and which other node devices each nodedevice is sending probe packets too. As the network may be configuredsuch that each node device is performing the probing function for eachlink connected to the node device, this monitoring function wouldprovide topology unit 76 with a complete representation of the variousconnections within the network, and topology unit 76 may use graphtheory to derive the topological representation of the various nodedevices and links within the network.

For each of the plurality of links, probe management unit 77 may select,based on the topological representation of the network, a node device ofthe two node devices connected by the respective link to measure one ormore QoE metrics for the respective link. As a result, the non-selectednode device does not measure the QoE metrics for the respective link,meaning that only the selected node device will perform the active orsynthetic probing functions for the respective link.

In selecting the node device of the two node devices connected by therespective link to measure the QoE metrics for the respective link,probe management unit 77 may instruct the selected node device of thetwo node devices connected by the respective link to send one or moreprobe packets over the respective link. In other examples, the nodedevices may automatically be configured to send the one or more probepackets over the respective links for the node devices. In suchinstances, rather than actively instructing the selected node devices tomeasure the QoE metrics for the respective link, probe management unit77 may instead instruct the non-selected node device of the two nodedevices to refrain from sending any additional probe packets over therespective link, thereby actively instructing the non-selected node tonot measure the QoE metrics.

In an effort to optimize the efficiency of the overall system describedherein, probe management unit 77 may select the node devices such thatthe probing processes are as evenly distributed across the node devicesas possible. In other words, if two node devices are connected by alink, the node device of the two node devices that is responsible forprobing fewer links may be better situated to handle the probing for thecurrent link. As such, when selecting the node devices for probing onrespective links, probe management unit 77 may perform an optimizationalgorithm. For each node device of the plurality of node devices, probemanagement unit 77 may examine each link of the plurality of links thatincludes the respective node device. For each link of the plurality oflinks that includes the respective node device probe management unit 77may determine whether the second node device connected to the respectivenode device by the respective link is configured to probe a total numberof links less than or equal to a total number of links being probed bythe respective node device. Responsive to determining that the secondnode device connected to the respective node device by the respectivelink is sending the total number of probes less than or equal to thetotal number of links being probed by the respective node device, probemanagement unit 77 may select the respective node device to refrain fromsending further probe packets to the second node device connected to therespective node device by the respective link over the respective link.

Probe management unit 77 may continue iterating through the node devicesand links in this manner until every link has only a single selectednode device, or until the system is optimized and cycling through thenode devices in this manner would not alter the overall probing plan.For instance, while there exist links in the plurality of links forwhich each node device connected by the respective link is sending aprobe packet over the respective link, probe management unit 77 mayselect a singular node device of the node devices connected by therespective link refrain from probing the respective link based on thetotal number of links being probed by each node device connected by therespective link.

In determining which node device is probing fewer links, probemanagement unit 77 may use the topological representation in the form ofan adjacency matrix. When probe management unit 77 selects the nodedevice of the two node devices connected by the respective link, probemanagement unit 77 may determine, based on the adjacency matrix thatindicates which node devices of the plurality of node devices aresending probe packets and how many links each node device is measuring arespective set of one or more QoE metrics for, which node device of thetwo node devices is measuring the respective sets of one or more QoEmetrics for fewer links. Probe management unit 77 could determine thisby calculating a sum of a column/row in the matrix for associated withthe node devices in question and comparing the sums. Once probemanagement unit 77 selects one of the node devices, probe managementunit 77 may update the adjacency matrix such that the sum of column/rowfor the non-selected node would be reduced by 1.

Although the above functions may be part of an initialization techniqueat the beginning of implementing the techniques described herein, thesetechniques may be repeated when new node devices enter the network.SD-WANs are dynamic in nature, and node devices may enter or leave thenetwork at any time. This means that links are created and removed fromthe system throughout the existence of the SD-WAN. As such, topologyunit 76 and probe management unit 77 may be configured to repeat theprocesses described herein whenever a new node device enters the systemor whenever a current node device leaves the system.

For instance, topology unit 76 may determine that a new node device hasentered the network, adding the new node to topology data 78. By thevery nature of entering the network, the new node device may beconnected to at least one node device of the plurality of node devicesvia at least one new link. For each new link of the at least one newlink, probe management unit 77 may determine whether the node deviceconnected to the new node device by the respective new link isconfigured to probe a total number of links less than or equal to thetotal number of links being probed by the new node device. Responsive todetermining that the node device connected to the new node device by therespective new link is probing the total number of links less than orequal to the total number of links being probed by the new node device,probe management unit 77 may select the new node device to refrain fromsending further probe packets to the node device connected to the newnode device by the respective new link over the respective new link.Conversely, responsive to determining that the node device connected tothe new node device by the respective new link is probing the totalnumber of links greater than the total number of links being probed bythe new node device, probe management unit 77 may select the node deviceconnected to the new node device over the respective new link to refrainfrom sending further probe packets to the new node device over therespective new link. Similar repetitions of the optimization functionsmay be applied when node devices leave the network, as some node devicesthat remain in the network may be more greatly effected by a node deviceleaving than others due to the links associated with the exiting nodedevice.

In response to selecting the selected node device to measure the one ormore QoE metrics for the respective link, probe management unit 77 mayreceive, from the selected node device, a set of one or more QoE metricsfor the respective link. This set of one or more QoE metrics wouldindicate QoE metrics for data flows flowing from the selected nodedevice to the non-selected node device. Probe management unit 77 maystore the set of QoE metrics for the respective link in metric database79, such that the QoE metrics for this link may be referenced in furtherextrapolations and estimations for other links and/or the reversedirection for the link. For instance, probe management unit 77 maydetermine, based on the set of one or more QoE metrics for therespective link, a set of one or more counter QoE metrics indicating QoEmetrics for data flows flowing from the non-selected node device to theselected node device.

While the above describes probe management unit 77 using the QoE metricsfor a link to estimate the reverse direction QoE metrics for the link,probe management unit 77 may use these QoE metrics for otherdeterminations as well. For instance, probe management unit 77 may usethe determined QoE metrics for each of the links to determine QoEmetrics for more complex paths that include multiple links, e.g., fornodes that are not directly connected to one another. Probe managementunit 77 may determine a path that includes a combination of a first linkof the plurality of links and a second link of the plurality of links,where the first link connects a first node device of the plurality ofnode devices and a second node device of the plurality of node devices.The second link connects the second node device of the plurality of nodedevices and a third node device of the plurality of node devices. Asthis path may be the shortest path between the first node device and thethird node device, the first node device of the plurality of nodedevices and the third node device of the plurality of node devices maynot be directly connected by any link of the plurality of links. Assuch, probe management unit 77 may retrieve a set of one or more QoEmetrics for the first link and a set of one or more QoE metrics for thesecond link, and use these retrieved QoE metrics to estimate a set ofone or more QoE metrics for the path.

Probe management unit 77 may also extend these techniques for logicalpaths connecting two node devices that are already connected by a link.For instance, for a first link of the plurality of links that connects afirst node device of the plurality of node devices and a second nodedevice of the plurality of node devices, probe management unit 77 mayestimate, based on the set of one or more QoE metrics for the first linkstored in metrics database 79, a set of one or more QoE metrics for alogical path connecting the first node device and the second nodedevice. While the logical path may be a different connection than thefirst link, similar physical hardware may be used in this connectionbetween the two node devices. As such, QoE metrics for the logical pathmay ultimately be similar to the QoE metrics for the measured link. Assuch, while the measurement may not be exact, this estimation may stillprovide reliable QoE metrics for the logical path without usingadditional resources to send additional probe packets over the logicalpath.

Device manager 75 may generate vendor-agnostic device information basedon the inputs provided by topology database 78 and metrics database 79.Vendor-agnostic device information may also be referred to as deviceabstract configuration information. Vendor-agnostic device informationis agnostic, per-device configuration information for each individualnetwork device in a network. In some examples, vendor-agnostic deviceinformation may comprise Extensible Markup Language (XML) schema or YetAnother Next Generation (YANG) schema information.

SDN controller 60 may then utilize device manager 75 to implement one ormore translation processes to translate vendor-agnostic deviceinformation into vendor-specific device information. Vendor-specificdevice information may also be referred to as vendor specific, orconcrete, device configuration information. Each individual networkdevice in the network may have both vendor-agnostic device informationand vendor-specific device information. In some examples,vendor-specific device information may be customizable via the use,e.g., of Jinja2 templates for each different vendor.

As a result, device manager 75 may be configured to generate bothvendor-agnostic device information and vendor-specific deviceinformation in the manner illustrated in FIG. 4. Vendor-agnostic deviceinformation may include vendor-agnostic device configuration and/orcommand information (e.g., entered via CLI 75 or other managementinterface), and vendor-specific device information may includevendor-specific device configuration and/or command information, as well(e.g., sent via network interface 66 and/or via protocols 70). SDNcontroller 60 illustrated in FIG. 4 may send vendor-specific deviceinformation to individual network devices for which the vendor-specificdevice information has been generated. For example, SDN controller 60may provide vendor-specific device configuration information to anetwork device via a configuration interface, and may providevendor-specific device command information to a network device via acommand interface. In this example, vendor-specific device configurationinformation and vendor-specific device command information may each beincluded in vendor-specific device information, which is translated fromvendor-agnostic device information. Additional information regardingtranslation of high-level configuration instructions to low-level deviceconfiguration can be found in U.S. patent application Ser. No.15/198,657, filed Jun. 30, 2016, and entitled TRANSLATING HIGH-LEVELCONFIGURATION INSTRUCTIONS TO LOW-LEVEL DEVICE CONFIGURATION, which ishereby incorporated by reference.

These techniques may enable probe management unit 77 to create a dynamicrepresentation of the QoE metrics for each link and each possibleconnection, in all directions, in topology database 78 and metricsdatabase 79. For instance, probe management unit 77 may store thereceived measurements for each link in metrics database 79. Then, foreach extrapolation/estimation (e.g., for each reverse direction of thereceived QoE metrics, for each logical path connecting two node devicesalso connected by a link, and for each complex path that is a shortestpath between two node devices not connected directly by a link)indicated in topology database 78, probe management unit 77 may storethe respective estimated QoE metrics in metrics database 79. Thisinformation may be updated as new metrics are received by probemanagement unit 77. With this information, whenever SDN controller 60must make a routing decision for application traffic or for trafficbetween nodes, SDN controller 60 may reference topology database 78 andthe QoE metrics stored within metrics database 79 to make routingdecisions for the traffic within the network.

There may be one or more advantages to using the techniques describedherein. While the actual metrics may not be exact in all situations, aparticular link will generally have similar QoE metrics in eachdirection. Similarly, when sending probe packets over a complex pathconsisting of multiple links, while the QoE metrics may not exactly be acombination of the links that make up the complex path, taking acombination of the QoE metrics across each link may still provideaccurate estimates of the QoE metrics for the path. As such, estimatingthe QoE metrics for various links between node devices in the mannerdescribed by the techniques of this disclosure may greatly reduce thebandwidth and other resources consumed by the node devices in an effortto reliably generate QoE metrics for the entire network. As node deviceshave fewer probe packets to generate and process, this may also increasethe efficiency of each individual node device, including reducingcentral processing unit (CPU) usage, memory using, and power consumptionfor each individual node device.

FIG. 4 is a block diagram illustrating an example network deviceconfigured to execute one or more functions to perform edge-basedrouting techniques, in accordance with the techniques of thisdisclosure. While the network device may be any network deviceconfigured to perform the techniques described herein, the networkdevice may be an example of SD-WAN appliance 18 of FIG. 1 or SD-WANappliance 38 of FIG. 2. SD-WAN appliance 80 may be described hereinwithin the context of SD-WAN system 2 of FIG. 1, and may represent anyof routers 8 or SD-WAN appliance 18, for example. Moreover, whiledescribed with respect to a particular network device, e.g., a router orSD-WAN appliance, the techniques may be implemented by any networkdevice that may operate as an SD-WAN appliance, such as a client device,a Layer 3 (L3) or L2/L3 switch, or server.

In this example, SD-WAN appliance 80 is divided into three logical orphysical “planes” to include a control plane 81 that performs controloperations for the device, a data plane 85 for forwarding transitnetwork traffic and a service plane 83 for application of one or morenetwork services 87 to transit packet flows that are forwarded by therouter. That is, router 81 implements three separate functionalities(e.g., the routing/control, forwarding data and network servicefunctionalities), either logically, e.g., as separate software instancesexecuting on the same set of hardware components, or physically, e.g.,as separate physical dedicated hardware components that eitherstatically implement the functionality in hardware or dynamicallyexecute software or a computer program to implement the functionality.In this example, a high-speed internal switch fabric 105 couples controlplane 81, service plane 83, and data plane 85 to deliver data units andcontrol messages among the units. Switch fabric 105 may represent aninternal switch fabric or cross-bar, bus, or link.

In the example of FIG. 4, control plane 81 includes control unit 82having master microprocessor(s) 102, which executes device managementservices, subscriber authentication and control plane routingfunctionality of SD-WAN appliance 80. Microprocessor 102 may compriseone or more general- or special-purpose processors such as a digitalsignal processor (DSP), an ASIC, a field programmable gate array (FPGA),or any other equivalent logic device. Accordingly, the terms “processor”or “controller,” as used herein, may refer to any one or more of theforegoing structures or any other structure operable to performtechniques described herein. Executables, such as probing engine 110,may be operable by microprocessor 102 to perform various actions,operations, or functions of SD-WAN appliance 80. For example,microprocessor 102 of SD-WAN appliance 80 may retrieve and executeinstructions stored by various data stores that cause microprocessor 102to perform the operations of probing engine 110.

One or more storage components (e.g., RIB 104) within SD-WAN appliance80 may store information for processing during operation of SD-WANappliance 80 (e.g., SD-WAN appliance 80 may store data accessed byprobing engine 110 during execution at SD-WAN appliance 80). In someexamples, the storage component is a temporary memory, meaning that aprimary purpose of the storage component is not long-term storage.Storage components on SD-WAN appliance 80 may be configured forshort-term storage of information as volatile memory and therefore notretain stored contents if powered off. Examples of volatile memoriesinclude random access memories (RAM), dynamic random access memories(DRAM), static random access memories (SRAM), and other forms ofvolatile memories known in the art.

Storage components, in some examples, also include one or morecomputer-readable storage media. Storage components in some examplesinclude one or more non-transitory computer-readable storage mediums.Storage components may be configured to store larger amounts ofinformation than typically stored by volatile memory. Storage componentsmay further be configured for long-term storage of information asnon-volatile memory space and retain information after power on/offcycles. Examples of non-volatile memories include magnetic hard discs,optical discs, floppy discs, flash memories, or forms of electricallyprogrammable memories (EPROM) or electrically erasable and programmable(EEPROM) memories. Storage components may store program instructionsand/or information (e.g., data) associated with probing engine 110.Storage components 248 may include a memory configured to store data orother information associated with probing engine 110.

In some examples, probing engine 110 may be configured to perform one ormore aspects of the various techniques described herein. As shownherein, probing engine 110 may be implemented as a component of controlunit 82 in the control plane. However, in other implementations, probingengine 110 may be a standalone component of a network device distinctand physically separate from SD-WAN appliance 80 and outside of any ofthe planes defined in SD-WAN appliance 80, assessing the received datatraffic outside of the control plane, data plane, and service plane. Insuch instances, control unit 82 may replicate application packets asthey are received, sending the original instances of the applicationtraffic to one of forwarding units 112 and sending the replicatedapplication traffic to probing engine 110 to evaluate separately fromthe regular data flow.

In general, control unit 82 represents hardware or a combination ofhardware and software of control that implements control plane protocols89A-89N (“routing protocols 89”) to learn and maintain routinginformation within routing information base 104 (“RIB 104”). RIB 104 mayinclude information defining a topology of a network, such as serviceprovider network of FIG. 1. Routing protocols 89 interact with kernel100 (e.g., by way of API calls) executing on control unit 82 to updateRIB 104 based on routing protocol messages received by SD-WAN appliance80. Kernel 100 may resolve the topology defined by routing informationin RIB 104 to select or determine one or more routes through thenetwork. For example, the kernel may generate forwarding information inthe form of forwarding information bases 103A-103N (“FIBs 103”) based onthe network topology represented in RIB 104, i.e., perform routeresolution. Typically, kernel 100 generates FIBs 103 in the form ofradix or other lookup trees to map packet information (e.g., headerinformation having destination information and/or a label stack) to nexthops and ultimately to interface ports of IFCs 114 associated withrespective forwarding units 112. Each of FIBs 103 may associate, forexample, network destinations with specific next hops and correspondingIFCs 114. For MPLS-related traffic forwarding, FIBs 103 stores, for agiven FEC, label information that includes an incoming label, anoutgoing label, and a next hop for a packet. Control unit 82 may thenprogram forwarding units 112 of data plane 85 with FIBs 103, whichinstalls the FIBs within lookup ASICs 106.

Data plane 85, in this example, is a decentralized data plane in thatdata plane functionality and packet forwarding functionality isdistributed among a plurality of forwarding units 112A-112N (“forwardingunits 112”). In the example of SD-WAN appliance 80, data plane 85includes forwarding units 112 that provide high-speed forwarding ofnetwork traffic received by interface cards 114A-114N (“IFCs 44”) viainbound links 116A-116N to outbound links 118A-118N. Forwarding units112 may each comprise one or more packet forwarding engine (“PFE”)coupled to respective interface cards 114 and may represent, forexample, a dense port concentrator (DPC), modular port concentrator(MPC), flexible physical interface card (PIC) concentrator (FPC), oranother line card, for example, that is insertable within a chassis orcombination of chassis of SD-WAN appliance 80.

As shown in the example of FIG. 3, each of forwarding units 112 includesa respective one of lookup ASICs 106A-106N (“lookup ASICs 106”) thatreceives control and data session traffic via IFC cards 114, performsroute lookups and, based on routes installed to FIBs 103, forwards thetraffic either to control unit 82 (control traffic destined for SD-WANappliance 80) or to one of forwarding units 40 (transit data traffic)for output via an interface to one of output links 48. In one example,lookup ASICs 106 are microcode-controlled chipsets programmablyconfigured by a slave microprocessor (not shown) executing on each offorwarding units 112. Specifically, in this example, each of ASICs 106may be controllable by internal microcode programmed by a slavemicroprocessor.

When forwarding packets, control logic within each lookup ASICs 106traverses the respective FIB 103 and, upon reaching a FIB entry for thepacket (e.g., a leaf node), the microcode-implemented control logicautomatically selects one or more forwarding next hops (FNHs) forforwarding the packet. In this way, ASICs 106 of forwarding units 112process packets by performing a series of operations on each packet overrespective internal packet forwarding paths as the packets traverse theinternal architecture of SD-WAN appliance 80. Operations may beperformed, for example, on each packet based on any of a correspondingingress interface, an ingress PFE 114, an egress PFE 114, an egressinterface or other components of SD-WAN appliance 80 to which the packetis directed prior to egress, such as one or more service cards.Forwarding units 112 each include forwarding structures that, whenexecuted, examine the contents of each packet (or another packetproperty, e.g., incoming interface) and on that basis make forwardingdecisions, apply filters, and/or perform accounting, management, trafficanalysis, and load balancing, for example.

In one example, each of forwarding units 112 arranges forwardingstructures as next hop data that can be chained together as a series of“hops” along an internal packet forwarding path for the network device.In many instances, the forwarding structures perform lookup operationswithin internal memory of ASICs 106, where the lookup may be performedagainst a tree (or trie) search, a table (or index) search. Otherexample operations that may be specified with the next hops includefilter determination and application, or a rate limiter determinationand application. Lookup operations locate, within a lookup datastructure (e.g., a lookup tree), an item that matches packet contents oranother property of the packet or packet flow, such as the inboundinterface of the packet. The result of packet processing in accordancewith the operations defined by the next hop forwarding structure withinASICs 106 determines the manner in which a packet is forwarded orotherwise processed by forwarding units 112 from its input interface onone of IFCs 114 to its output interface on one of IFCs 114.

Lookup ASICs 106 may be implemented using forwarding applicationspecific integrated circuits (ASICs), field programmable gate arrays(FPGAs), or any other equivalent integrated or discrete logic circuitry,as well as any combinations of such components. Each of forwarding units112 may include substantially similar components to performsubstantially similar functionality.

Service plane 83 of SD-WAN appliance 80 includes a plurality of serviceunits 13A-113K (“service units 13”) that may be, as examples, removableservice cards, which are configured to apply network services to packetsflowing through data plane 85. That is, when forwarding packets,forwarding units 112 may steer packets to service plane 83 forapplication of one or more network services 131 by service units 113. Inthis example, each of service units 113 includes a microprocessor 127configured to execute hypervisor 129 to provide an operating environmentfor a plurality of network services 131. As examples, service units 113may apply firewall and security services, carrier grade network addresstranslation (CG-NAT), media optimization (voice/video), IPSec/VPNservices, deep packet inspection (DPI 131A), HTTP filtering, counting,accounting, charging, and load balancing of packet flows or other typesof services applied to network traffic. Each of services 131 may beimplemented, for example, as virtual machines or containers executed byhypervisor 129 and microprocessor 127.

In the example of FIG. 4, control unit 82 provides an operatingenvironment for probing engine 110. In some examples, control unit 82may use probing engine 110 to execute one or more TWAMP or RPM logicalroles, such as a control client, a server, a sessions sender, and asession reflector.

In accordance with the techniques described herein, probing engine 110may receive instructions from an SDN controller (e.g., SDN controller60) to probe, or to not probe, any of the one or more links thatconnects SD-WAN appliance 80 to another node device.

Probing engine 110 may follow the instructions received, eitherperforming a probing process to measure QoE metrics for a linkconnecting SD-WAN appliance 80 to another node device or refraining fromperforming the probing process. For the links that probing engine 110does measure QoE metrics for, probing engine 110 may send the measuredQoE metrics to the SDN controller. The SDN controller may aggregatethese metrics from SD-WAN appliance 80 and other node devices in thenetwork in accordance with the techniques described herein.

Probing engine 110 may also store the measured QoE metrics locally inreference database 90. Probing engine 110 may use the QoE metrics storedin reference database 90 and SLA metrics stored in SLA database 92 toassign application traffic flows to various links.

FIG. 5 is a conceptual flow diagram illustrating an example topology fora plurality of nodes 502A-502G in a software-defined wide area networksystem, in accordance with the techniques of this disclosure.Application probes play a vital role in measuring the performance of thenetwork and to continuously verify that the links are able to meet thedesired characteristics of the application packets. Probes could beactive (e.g., synthetic packets resembling the application data packets)or passive (e.g., embedded probe headers in the live traffic) in nature.While the passive probes do not result in additional traffic, thepackets need to be intercepted and processed, resulting in additionalload as well as concerns on privacy and security. Synthetic probingrelies on injecting test packets that mimics the actual application,thereby measuring the network performance. Network topology also is animportant factor in planning and management for the network.

In the example of FIG. 5, nodes 502 may each be devices in a networkconfigured to transmit data amongst one another over various links. Forinstance, nodes 502 may each be separate pieces of customer equipment inan access network (e.g., CE 11 in access network 6 of FIG. 1), separatesubscriber devices within a network (e.g., subscriber devices 16 of FIG.1), nodes in a data center (e.g., nodes 10 of data center 9 of FIG. 1),separate SD-WAN appliances in a SD-WAN (e.g., SD-WAN appliance 18 ofFIG. 1), or any combination thereof in a SDN system (e.g., SD-WAN system2 of FIG. 1). As such, FIG. 5 may be an example of a simplified networktopology of SD-WAN 7 of FIG. 1, access network 6 of FIG. 1, or SD-WANsystem 2 of FIG. 1.

An example of the techniques described herein to avoid redundant probesand probe duplication to reduce the traffic is described below withrespect to FIG. 5. Consider the topology of node 502A and node 502B. Ifa probe is being sent from node 502A to node 502B, an SDN controller(e.g., SDN controller 14) may avoid running another probe from node 502Bto node 502A and instead correlate the metrics collected on node 502A totake decisions on node 502B as well.

The SDN controller may also ensure uniform distribution across thenetwork topology. In case of a complex full mesh, if there are N devicesin full mesh, the worst case is that each device will run N−1 probes(N(N−1) probes in total) for a given protocol. Using the techniquesdescribed herein, this can be optimized to N(N−1)/2 total probes, andadditionally distribute the load such that each device runs (N−1)/2probes.

The techniques described herein also grant the SDN controller theability to perform complex probing. Suppose the cloud controllerreceives configuration data requesting to determine QoE metrics for apath from node 502A to node 502C. If the cloud controller has alreadydetermined the QoE metrics between node 502A to node 502B and from node502B to node 502C, the cloud controller may simply aggregate the QoE oneach of these links to extrapolate the network behavior between node502A to node 502C. The solution provided by the techniques describedherein may assume device support for networking probes (such as RPM,application level probes etc.), as it may leverage existingconfiguration interfaces.

TABLE I Adjacency Matrix(assuming single probe) A B C D F F G A 0 1 0 00 0 0 B 1 0 1 1 0 0 1 C 0 1 0 0 1 1 0 D 0 1 0 0 1 1 0 E 0 0 1 1 0 0 1 F0 0 1 1 0 0 1 G 0 1 0 0 1 1 0

Table I. Adjacency Matrix (Assuming Single Probe)

TABLE II Raw distribution of probes from each of the Nodes A B C D F F G1 4 3 3 3 3 3

Table II. Raw Distribution of Probes From Each of the Nodes

TABLE III Optimized distribution A B C D E F G 1 2 2 1 2 1 1

In an example of the distribution algorithm used in the techniquesdescribed herein, the cloud controller (e.g., SDN controller 14) mayiterate through each of the nodes. For every node (e.g., node 502A), thecloud controller may loop through the adjacency nodes. As the top halfof matrix is already optimized, the cloud controller may only look atthe sub matrix below this node. If the adjacency node has equal or lessload, the cloud controller may determine to not probe it. Otherwise, thecloud controller may clear the probe flag on the adjacent node. Thecloud controller may continue to prune the probes from the device basedon the re-distribution. In this way, the techniques described herein mayutilize graph theory concepts in deriving the optimal distribution ofprobes throughout the system. In the case of multiple probes, the countmay be reflected in an adjacency matrix.

TABLE IV Sub-Matrix for Node A A B C D E F G A 0 1 0 0 0 0 0 B 1 0 1 1 00 1 C 0 1 0 0 1 1 0 D 0 1 0 0 1 1 0 E 0 0 1 1 0 0 1 F 0 0 1 1 0 0 1 G 01 0 0 1 1 0

TABLE V Sub-Matrix for Node B A B C D E F G A 0 1 0 0 0 0 0 B 0 0 0 1 00 1 C 0 1 0 0 1 1 0 D 0 0 0 0 1 1 0 E 0 0 1 1 0 0 1 F 0 0 1 1 0 0 1 G 01 0 0 1 1 0

TABLE VI Sub-Matrix for Node C A B C D E F G A 0 1 0 0 0 0 0 B 0 0 0 1 00 1 C 0 1 0 0 0 1 0 D 0 0 0 0 1 1 0 E 0 0 1 1 0 0 1 F 0 0 1 1 0 0 1 G 01 0 0 1 1 0

TABLE VII Sub-Matrix for Node E A B C D E F G A 0 1 0 0 0 0 0 B 0 0 0 10 0 1 C 0 1 0 0 0 1 0 D 0 0 0 0 0 1 0 E 0 0 0 1 0 0 1 F 0 0 0 0 0 0 1 G0 0 0 0 1 0 0

The SDN controller may capture the user intent (e.g., to measure UDPperformance across the network via synthetic probes). This architecturemay also analyze the network topology and build a map of the currenttopology, all while utilizing existing probes. This architecture mayreduce redundancy by using a single measurement to extrapolate QoEmetrics for reverse paths and additional logical paths (e.g., VPNs,etc.) originating from the same device. The SDN controller may prune theprobes to avoid duplicates and redundancy (e.g., device on the reversepath need not probe again). The SDN controller may also shuffle theprobes to ensure the uniform distribution across the topology (or thesub-set within the topology). This architecture may leverage theexisting interface on the device to configure and monitor the probes.The SDN controller may further maintain this intent map to correlate theprobe related metrics (e.g., run a single probe between nodes 502A and502B of FIG. 5 and the metrics such as jitter, packet loss, etc. may beapplicable for both A and B).

Through utilizing the techniques described herein, the SDN controllermay provide a unified management system for networking probes, enablinga network administrator to better visualize the state of network to helpaid in better planning. For instance, once the SDN controller constructsthe topological representation of the network, the SDN controller mayoutput a visualization of the topological representation for display toa user. The SDN controller may also output a visualization of theresults of the probe management process, indicating which node devicesare sending probe packets and over which links those node devices aresending probe packets. This may enable the user to make more informeddecisions about different routing possibilities, which node devices areprobing the most devices, and how else the user may change theconfigurations or intent to best optimize the resources on the network.

FIG. 6 is conceptual diagram illustrating an example architecture of acloud controller 602 configured to perform edge-based routingtechniques, in accordance with the techniques of this disclosure. Cloudcontroller 602 may be an example of SDN controller 14 of FIG. 1, and maybe configured to perform any of the techniques described herein. In someexamples, cloud controller 602 may dynamically redistribute the probeassignments as nodes enter or leave the topology. The techniquesdescribed herein may also be utilized for other network arrangementsthan those described herein, including hub and spoke networks.

In the example of FIG. 6, devices 612A and 612B are connected bymultiple logical paths under different protocols, including link controlmanagement protocol (LCMP), Voice over Internet protocol (VOIP),transmission control protocol (TCP), and UDP. As shown in FIG. 6,devices 612A and 612B may be exchanging probe packets over each of thelogical paths to determine QoE metrics for each of the logical paths.The probe packets may be generated and analyzed by various components ofeach of devices 612A and 612B, such as agents 614A and 614B,respectively.

In accordance with the techniques described herein, cloud controller 602may communicate with devices 612A and 612B via configuration interfaces610 to alter the probing protocol for these devices. For instance, cloudcontroller 602 may instruct device 612A to refrain from sending anyfurther probe packets to 612B. Further, cloud controller 602 mayinstruct device 612B to only send a probe packet over the UDP logicalpath. Cloud controller 602 may receive the QoE metrics from device 612Band store the QoE metrics in database 606. Cloud controller 602 may thenextrapolate this data to estimate the QoE metrics for the UDP logicalpath between flowing from device 612A to device 612B, as well as eachother logical path (LCMP, VOIP, and TCP) bidirectionally. Cloudcontroller 602 may also store these estimated QoE metrics in database606.

As such, rather than sending 4, or even 8, probe packets to evaluate theQoE metrics between the links connecting devices 612A and 612B, cloudcontroller 602 may determine reliable QoE metrics for each path betweendevices 612A and 612B by sending only a single probe packet over thelink. As such, the techniques described herein may reduce the trafficcaused by the probe packets by as much as 87.5%.

FIG. 7 is a flow diagram illustrating an example technique for asoftware-defined wide area network system that performs probe assignmentfunctions based on a topological representation of the network, inaccordance with the techniques of this disclosure. The example operationmay be performed by SDN controller 14 from FIG. 1, or SDN controller 60of FIG. 3, for example. The following are steps of the process, althoughother examples of the process performed in the techniques of thisdisclosure may include additional steps or may not include some of thebelow-listed steps.

In accordance with the techniques described herein, SDN controller 14may construct, for SD-WAN system 2 that includes a plurality of nodedevices (e.g., subscriber devices 16 and service nodes 10), atopological representation of SD-WAN system 2 (702). The topologicalrepresentation may take any form (e.g., a matrix, a database, a graphic,text, or any other data structure) that provides an indication of eachof the node devices and an indication of each link of a plurality oflinks, where each link connects two of the node devices. SDN controller14 may select a link to evaluate for QoE metric measurement (704).

For this respective link, SDN controller 14 may select, based on thetopological representation of the network, a node device of the two nodedevices connected by the respective link to measure one or more QoEmetrics for the respective link (706). As a result, the non-selectednode device does not measure the QoE metrics for the respective link,meaning that only the selected node device will perform the active orsynthetic probing functions for the respective link.

In response to selecting the selected node device to measure the one ormore QoE metrics for the respective link, SDN controller 14 may receive,from the selected node device, a set of one or more QoE metrics for therespective link (708). These QoE metrics may be based on a received userintent, in the sense that the intent may specify what information is tobe measured or what decision will be made based on the receivedinformation. As such, when selecting the node device of the two nodedevices, SDN controller 14 may indicate what QoE metrics are to bemeasured by the selected node device. This set of one or more QoEmetrics would indicate QoE metrics for data flows flowing from theselected node device to the non-selected node device. SDN controller 14may store the set of QoE metrics for the respective link in a database(710), such that the QoE metrics for this link may be referenced infurther extrapolations and estimations for other links and/or thereverse direction for the link. For instance, SDN controller 14 maydetermine, based on the set of one or more QoE metrics for therespective link, a set of one or more counter QoE metrics indicating QoEmetrics for data flows flowing from the non-selected node device to theselected node device (712).

SDN controller 14 may repeat this process for each link. For instance,if additional links exist where SDN controller 14 must assign a nodedevice to measure the QoE metrics for that link (YES branch of 714), SDNcontroller 14 may select such a link for QoE evaluation, as describedabove (704). Otherwise (NO branch of 714), SDN controller 14 may monitorthe network and assign application traffic during the normal course ofoperation.

FIG. 8 is a flow diagram illustrating an example technique for asoftware-defined wide area network system that performs probe assignmentfunctions based on a topological representation of the network, inaccordance with the techniques of this disclosure. The example operationmay be performed by SDN controller 14 from FIG. 1, or SDN controller 60of FIG. 3, for example. The following are steps of the process, althoughother examples of the process performed in the techniques of thisdisclosure may include additional steps or may not include some of thebelow-listed steps.

In accordance with the techniques described herein, topology unit 76 mayreceive configuration data indicative of a user intent for measuringapplication quality of experience (QoE) in a network (802). Based onthis intent, topology unit 76 may construct a topological representationof the network that contains or is managed by SDN controller 60 (804).The topological representation of the network indicates each of aplurality of node devices in the network and an indication of each linkof a plurality of links, each link connecting two node devices of theplurality of node devices. Probe management unit 77 may then modify,from an initial probing list that indicates which node devices in theplurality of node devices are performing a probing process and overwhich links said node devices are performing the probing process, one ormore entries from the probing list to create a modified probing listsuch that only a single node device is performing the probing processfor each of the links (806). Using this modified probing list, probemanagement unit 77 may instruct the node devices to perform probingprocesses on links corresponding only to the entries in the modifiedprobing list (808). These instructions include instructing said nodedevices to send QoE metric data to SDN controller 60 upon performing theprobing process. In essence, SDN controller 60 also instructs, eitherimplicitly or explicitly, other node devices to refrain from performingthe probing process over particular links if the corresponding entrieswere pruned from the probing list. The probing list may also be based onthe user intent in the way that the probing list may include whatinformation is to be measured during the probing process on each device.Once the node devices send the requested QoE metric data to SDNcontroller 60, topology unit 76 may aggregate the QoE metric data tocomplete the topology by extrapolating the received QoE metric data toestimate QoE metrics for connections not explicitly included in theprobing list (810). In completing the topology, topology unit 76 mayhave a complete map of the node devices in the network, the linksconnecting the various node devices in the network, and application QoEmetrics for each connection between node devices, in each direction.Using this complete topology, SDN controller 60 may orchestrate dataflows within the network to satisfy the configuration data indicative ofthe user intent.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof. Various featuresdescribed as modules, units or components may be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices or other hardware devices. In some cases, various featuresof electronic circuitry may be implemented as one or more integratedcircuit devices, such as an integrated circuit chip or chipset.

If implemented in hardware, this disclosure may be directed to anapparatus such as a processor or an integrated circuit device, such asan integrated circuit chip or chipset. Alternatively or additionally, ifimplemented in software or firmware, the techniques may be realized atleast in part by a computer-readable data storage medium comprisinginstructions that, when executed, cause a processor to perform one ormore of the methods described above. For example, the computer-readabledata storage medium may store such instructions for execution by aprocessor.

A computer-readable medium may form part of a computer program product,which may include packaging materials. A computer-readable medium maycomprise a computer data storage medium such as random access memory(RAM), read-only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),Flash memory, magnetic or optical data storage media, and the like. Insome examples, an article of manufacture may comprise one or morecomputer-readable storage media.

In some examples, the computer-readable storage media may comprisenon-transitory media. The term “non-transitory” may indicate that thestorage medium is not embodied in a carrier wave or a propagated signal.In certain examples, a non-transitory storage medium may store data thatcan, over time, change (e.g., in RAM or cache).

The code or instructions may be software and/or firmware executed byprocessing circuitry including one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, functionality described in this disclosure may be providedwithin software modules or hardware modules.

Various examples have been described. These and other examples arewithin the scope of the following claims.

The invention claimed is:
 1. A method comprising: constructing, by anetwork device for a network that includes a plurality of node devices,a topological representation of the network, wherein the topologicalrepresentation comprises an indication of each of the plurality of nodedevices and an indication of each link of a plurality of links, eachlink connecting two node devices of the plurality of node devices; andfor each of the plurality of links: selecting, by the network device andbased on the topological representation of the network, a node device ofthe two node devices connected by the respective link to measure one ormore quality of experience (QoE) metrics for the respective link,wherein the non-selected node device does not measure the QoE metricsfor the respective link; in response to selecting the selected nodedevice to measure the one or more QoE metrics for the respective link,receiving, by the network device and from the selected node device, aset of one or more QoE metrics for the respective link, wherein the setof one or more QoE metrics indicate QoE metrics for data flows flowingfrom the selected node device to the non-selected node device; storing,by the network device, the set of QoE metrics for the respective link ina database; and determining, by the network device and based on the setof one or more QoE metrics for the respective link, a set of one or morecounter QoE metrics indicating QoE metrics for data flows flowing fromthe non-selected node device to the selected node device.
 2. The methodof claim 1, further comprising: determining, by the network device, apath comprising a combination of a first link of the plurality of linksand a second link of the plurality of links, wherein the first linkconnects a first node device of the plurality of node devices and asecond node device of the plurality of node devices, wherein the secondlink connects the second node device of the plurality of node devicesand a third node device of the plurality of node devices, and whereinthe first node device of the plurality of node devices and the thirdnode device of the plurality of node devices are not directly connectedby any link of the plurality of links; retrieving, by the networkdevice, a set of one or more QoE metrics for the first link and a set ofone or more QoE metrics for the second link; and determining, by thenetwork device and based on the set of one or more QoE metrics for thefirst link and the set of one or more QoE metrics for the second link, aset of one or more QoE metrics for the path.
 3. The method of claim 1,further comprising: for a first link of the plurality of links thatconnects a first node device of the plurality of node devices and asecond node device of the plurality of node devices, determining, by thenetwork device, and based on the set of one or more QoE metrics for thefirst link stored in the database, a set of one or more QoE metrics fora logical path connecting the first node device and the second nodedevice, wherein the logical path is a different connection than thefirst link.
 4. The method of claim 1, further comprising: for each nodedevice of the plurality of node devices: for each link of the pluralityof links that includes the respective node device: determining, by thenetwork device, whether the second node device connected to therespective node device by the respective link is configured to probe atotal number of links less than or equal to a total number of linksbeing probed by the respective node device; and responsive todetermining that the second node device connected to the respective nodedevice by the respective link is sending the total number of probes lessthan or equal to the total number of links being probed by therespective node device, selecting, by the network device, the respectivenode device to refrain from sending further probe packets to the secondnode device connected to the respective node device by the respectivelink over the respective link.
 5. The method of claim 4, furthercomprising: while there exist links in the plurality of links for whicheach node device connected by the respective link is sending a probepacket over the respective link, selecting, by the network device, asingular node device of the node devices connected by the respectivelink refrain from probing the respective link based on the total numberof links being probed by each node device connected by the respectivelink.
 6. The method of claim 1, wherein selecting the node device of thetwo node devices connected by the respective link comprises determining,based on an adjacency matrix that indicates which node devices of theplurality of node devices are sending probe packets and how many linkseach node device is measuring a respective set of one or more QoEmetrics for, which node device of the two node devices is measuring therespective sets of one or more QoE metrics for fewer links, the methodfurther comprising in response to selecting the selected node device tosend to measure the one or more QoE metrics for the respective link,updating, by the network device, the adjacency matrix.
 7. The method ofclaim 1, further comprising: determining, by the network device, that anew node device has entered the network, wherein the new node device isconnected to at least one node device of the plurality of node devicesvia at least one new link; for each new link of the at least one newlink: determining, by the network device, whether the node deviceconnected to the new node device by the respective new link isconfigured to probe a total number of links less than or equal to thetotal number of links being probed by the new node device; responsive todetermining that the node device connected to the new node device by therespective new link is probing the total number of links less than orequal to the total number of links being probed by the new node device,selecting, by the network device, the new node device to refrain fromsending further probe packets to the node device connected to the newnode device by the respective new link over the respective new link; andresponsive to determining that the node device connected to the new nodedevice by the respective new link is probing the total number of linksgreater than the total number of links being probed by the new nodedevice, selecting, by the network device, the node device connected tothe new node device over the respective new link to refrain from sendingfurther probe packets to the new node device over the respective newlink.
 8. The method of claim 1, wherein selecting the node device of thetwo node devices connected by the respective link to measure the one ormore quality of experience (QoE) metrics for the respective linkcomprises: instructing, by the network device, the selected node deviceof the two node devices connected by the respective link to send one ormore probe packets over the respective link.
 9. The method of claim 1,wherein constructing the topological representation of the networkcomprises: monitoring, by the network device, one or more probe packetssent over each of the plurality of links; and constructing, by thenetwork device, based on the one or more probe packets sent over each ofthe plurality of links, and using graph theory, the topologicalrepresentation.
 10. A network device comprising: a memory; and one ormore processors in communication with the memory, the one or moreprocessors configured to: construct, for a network that includes aplurality of node devices, a topological representation of the network,wherein the topological representation comprises an indication of eachof the plurality of node devices and an indication of each link of aplurality of links, each link connecting two node devices of theplurality of node devices; and for each of the plurality of links:select, based on the topological representation of the network, a nodedevice of the two node devices connected by the respective link tomeasure one or more quality of experience (QoE) metrics for therespective link, wherein the non-selected node device does not measurethe QoE metrics for the respective link; in response to selecting theselected node device to measure the one or more QoE metrics for therespective link, receive, from the selected node device, a set of one ormore QoE metrics for the respective link, wherein the set of one or moreQoE metrics indicate QoE metrics for data flows flowing from theselected node device to the non-selected node device; store the set ofQoE metrics for the respective link in a database; and determine, basedon the set of one or more QoE metrics for the respective link, a set ofone or more counter QoE metrics indicating QoE metrics for data flowsflowing from the non-selected node device to the selected node device.11. The network device of claim 10, wherein the one or more processorsare further configured to: determine a path comprising a combination ofa first link of the plurality of links and a second link of theplurality of links, wherein the first link connects a first node deviceof the plurality of node devices and a second node device of theplurality of node devices, wherein the second link connects the secondnode device of the plurality of node devices and a third node device ofthe plurality of node devices, and wherein the first node device of theplurality of node devices and the third node device of the plurality ofnode devices are not directly connected by any link of the plurality oflinks; retrieve a set of one or more QoE metrics for the first link anda set of one or more QoE metrics for the second link; and estimate,based on the set of one or more QoE metrics for the first link and theset of one or more QoE metrics for the second link, a set of one or moreQoE metrics for the path.
 12. The network device of claim 10, whereinthe one or more processors are further configured to: for a first linkof the plurality of links that connects a first node device of theplurality of node devices and a second node device of the plurality ofnode devices, estimate, based on the set of one or more QoE metrics forthe first link stored in the database, a set of one or more QoE metricsfor a logical path connecting the first node device and the second nodedevice, wherein the logical path is a different connection than thefirst link.
 13. The network device of claim 10, wherein the one or moreprocessors are further configured to: for each node device of theplurality of node devices: for each link of the plurality of links thatincludes the respective node device: determine whether the second nodedevice connected to the respective node device by the respective link isconfigured to probe a total number of links less than or equal to atotal number of links being probed by the respective node device; andresponsive to determining that the second node device connected to therespective node device by the respective link is sending the totalnumber of probes less than or equal to the total number of links beingprobed by the respective node device, select the respective node deviceto refrain from sending further probe packets to the second node deviceconnected to the respective node device by the respective link over therespective link.
 14. The network device of claim 10, wherein the one ormore processors being configured to select the node device of the twonode devices connected by the respective link comprises the one or moreprocessors being configured to determine, based on an adjacency matrixthat indicates which node devices of the plurality of node devices aresending probe packets and how many links each node device is measuring arespective set of one or more QoE metrics for, which node device of thetwo node devices is measuring the respective sets of one or more QoEmetrics for fewer links, and wherein the one or more processors arefurther configured to, in response to selecting the selected node deviceto send to measure the one or more QoE metrics for the respective link,update the adjacency matrix.
 15. The network device of claim 10, whereinthe one or more processors are further configured to: determine that anew node device has entered the network, wherein the new node device isconnected to at least one node device of the plurality of node devicesvia at least one new link; for each new link of the at least one newlink: determine whether the node device connected to the new node deviceby the respective new link is configured to probe a total number oflinks less than or equal to the total number of links being probed bythe new node device; and responsive to determining that the node deviceconnected to the new node device by the respective new link is probingthe total number of links less than or equal to the total number oflinks being probed by the new node device, select the new node device torefrain from sending further probe packets to the node device connectedto the new node device by the respective new link over the respectivenew link; and responsive to determining that the node device connectedto the new node device by the respective new link is probing the totalnumber of links greater than the total number of links being probed bythe new node device, select the node device connected to the new nodedevice over the respective new link to refrain from sending furtherprobe packets to the new node device over the respective new link. 16.The network device of claim 11, wherein the network device comprises asoftware-defined networking (SDN) controller device.
 17. A methodcomprising: receiving, by a network device, configuration dataindicative of a user intent for measuring application quality ofexperience (QoE) in a network that includes a plurality of node devicesand a plurality of links, each link connecting two node devices of theplurality of node devices; constructing, by the network device, atopological representation of the network, wherein the topologicalrepresentation comprises an indication of each of the plurality of nodedevices and an indication of each link of the plurality of links;modifying, by the network device and based on the user intent and thetopological representation of the network, one or more entries from aninitial probing list to create a modified probing list, wherein theinitial probing list comprises an adjacency matrix having a plurality ofentries, wherein each entry indicates a particular node device from theplurality of node that is performing a probing process for a particularlink of the plurality of links, wherein modifying the one or moreentries from the initial probing list comprises: determining which nodedevice of the two node devices in a particular entry of the adjacencymatrix is measuring the respective application QoE metric data for fewerlinks; and pruning the entry from the initial probing list correspondingto a node device of the two node devices that is measuring therespective application QoE metric data for more links; instructing, bythe network device and in accordance with the modified probing list, oneor more node devices of the plurality of node devices to perform theprobing process on the respective link in the corresponding entry in themodified probing list, wherein the probing process generates applicationQoE metric data for the respective link; and in response to receivingthe application QoE metric data from the one or more node devicesinstructed to perform the probing process, aggregating, by the networkdevice, the application QoE metric data.
 18. The method of claim 17,wherein aggregating the application QoE metric data comprises:extrapolating, by the network device, the received application QoEmetric data to estimate application QoE metric data for each entry ofthe plurality of entries pruned from the initial probing list.
 19. Themethod of claim 17, further comprising: determining, by the networkdevice, that a new node device has entered the network, wherein the newnode device is connected to at least one node device of the plurality ofnode devices via at least one new link; and in response to determiningthat the new node has entered the network, for each new link of the atleast one new link: determining, by the network device, whether the nodedevice connected to the new node device by the respective new link isconfigured to probe a total number of links less than or equal to thetotal number of links being probed by the new node device; responsive todetermining that the node device connected to the new node device by therespective new link is probing the total number of links less than orequal to the total number of links being probed by the new node device,selecting, by the network device, the new node device to refrain fromsending further probe packets to the node device connected to the newnode device by the respective new link over the respective new link; andresponsive to determining that the node device connected to the new nodedevice by the respective new link is probing the total number of linksgreater than the total number of links being probed by the new nodedevice, selecting, by the network device, the node device connected tothe new node device over the respective new link to refrain from sendingfurther probe packets to the new node device over the respective newlink.